FIG. 1 is a block diagram illustrating non-limiting exemplary architecture of a distributed file system 100 implementing a Network Attached Storage (NAS) in accordance with the prior art. Distributed file server 120 may include a plurality of nodes (aka controllers) 130-1 to 130-x connected to a bus 180 operating in Internet Small Computer Systems Interface (iSCSI), a fiber channel (FC) or the like.
Bus 180 connects distributed file server 120 to a plurality of block storage devices 190 possibly configured as a part of a Storage Area Network (SAN) device aligned, for example, in a Redundant Array of Independent Disks (RAID) configuration.
Each of nodes 130-1 to 130-x may include a central processing unit (CPU) 160-1 to 160-x respectively, and memory units 150-1 to 150-x respectively, on which several processes are being executed. Nodes 130-1 to 130-x may communicate with a plurality of clients over network protocols such as Network File System (NFS) and Server Message Block (SMB).
Some of the processes running over nodes 130-1 to 130-x may include file system daemons (FSDs) 170-1 to 170-x. Each of nodes 130-1 to 130-x may include one or more FSDs which serve as containers for services and effectively control files in distributed file server 120.
Files in distributed file server 120 are distributed across FSDs 170-1 to 170-x and across nodes 130-1 to 130-x. Distributed file server 120 may also include file servers 140-1 to 140-x in at least one of nodes 130-1 to 130-x, wherein each of file servers 140-1 to 140-x may receive file system connect requests 112 from clients such as client machine 110.
Such client machine 110 may include, in a non-limiting example, Windows™ clients communicating over Server Message Block (SMB) protocol. Upon receiving such a connect request 112, file servers 140-1 to 140-x refer the requests to one of FSDs 170-1 to 170-x that holds the required file.
In accordance with the SMB protocol, each of FSDs 170-1 to 170-x may include an SMB server (not shown here) which is a process running on the node and configured to control the network communication in accordance with the SMB protocol. When a file is first requested by a client, for each client requesting a file, the SMB server allocates a unique persistent file handle associated with a specific client. The handle is an object which determines file name and other identifiers and the state which indicates the permissions of a specific client and other clients, and what other operations are the other clients are prevented from (aka share mode).
In SMB protocol, the handle serves as a mechanism allowing a specific client to establish or reestablish connection to a certain file after disconnection. In a case that SMB3 protocol is implemented, system failovers and/or node/FSD exchanges may be transparent to a client. That is, if connection with an FSD is lost, for example if a node or FSD fails or if the file server redirects a file to another node or FSD for better load balancing, the client may be reconnected to another node/FSD without interruption with the client's operations. In such cases, the persistent handle state must be preserved from the moment of connection loss and until client is reconnected again, for example to the other node/FSD.
The persistent handle state is notified about disconnection or reconnection of the client. However, in distributed environment such as distributed file server 120, the notifications might arrive in non-chronological order, which may cause a failure in case a former disconnection notice arrives to the handle state after the handle was already reconnected and a corresponding reconnection notice was received.